Worst-case Complexity of Cyclic Coordinate Descent: $O(n^2)$ Gap with Randomized Version

This paper concerns the worst-case complexity of Gauss-Seidel method for solving a positive semidefinite linear system; or equivalently, that of cyclic coordinate descent (C-CD) for minimizing a convex quadratic function. The known provable complexity of C-CD can be O(n) times slower than gradient descent (GD) and O(n) times slower than randomized coordinate descent (R-CD). However, these gaps seem rather puzzling since so far they have not been observed in practice; in fact, C-CD usually converges much faster than GD and sometimes comparable to R-CD. Thus some researchers believe the gaps are due to the weakness of the proof, but not that of the C-CD algorithm itself. In this paper we show that the gaps indeed exist. We prove that there exists an example for which CCD takes at least Õ(nκ) or Õ(nκCD) operations, where κ is the condition number, κCD is a well-studied quantity that determines the convergence rate of R-CD, and Õ(·) hides the dependency on log 1/ . This implies that C-CD can indeed be O(n) times slower than GD, and O(n) times slower than R-CD in the worst case. Our result establishes one of the few examples in continuous optimization that demonstrates a large gap between a deterministic algorithm and its randomized counterpart. Based on the example, we establish several almost tight complexity bounds of C-CD for quadratic problems. One difficulty with the analysis of the constructed example is that the spectral radius of a non-symmetric iteration matrix does not necessarily constitutes a lower bound for the convergence rate.